Instructional support platform for interactive learning environments

ABSTRACT

In various embodiments, subject matter for improving discussions in connection with an educational resource is identified and summarized by analyzing annotations made by students assigned to a discussion group to identify high-quality annotations likely to generate responses and stimulate discussion threads, identifying clusters of high-quality annotations relating to the same portion or related portions of the educational resource, extracting and summarizing text from the annotations, and combining, in an electronically represented document, the extracted and summarized text and (i) at least some of the annotations and the portion or portions of the educational resource or (ii) clickable links thereto.

RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. ProvisionalPatent Application No. 62/261,387, filed Dec. 1, 2015, U.S. ProvisionalPatent Application No. 62/261,397, filed Dec. 1, 2015, U.S. ProvisionalPatent Application No. 62/261,398, filed Dec. 1, 2015, and U.S.Provisional Patent Application No. 62/261,400, filed Dec. 1, 2015, theentire disclosure of each of which is hereby incorporated herein byreference.

TECHNICAL FIELD

In various embodiments, the present invention relates generally toonline learning, and in particular to resources for enhancing andpersonalizing learning experiences involving an online component.

BACKGROUND

As digital textbooks inexorably replace traditional printed media, andonline social resources such as discussion boards supplement classroominstruction, teachers and publishers are finding new opportunities forengaging students. Students with access to digital materials mayannotate a shared digital version of a class text or videos, ask andanswer each other's questions, and interact with the teaching staffwhile reading. The advantages are substantial: instead of waiting daysuntil office hours to get past a conceptual roadblock, students can aska question at any time and often get a response within minutes. Studentmotivation is enhanced through online interactions that enable them toshare interest and knowledge.

In increasing number of classrooms, when students are given readingmaterial as homework assignments, it is in digital format and they areallowed to highlight a passage and add a comment or question. Otherstudents (and the teaching staff) can then see this immediately and cananswer questions or add their own comments (in an interaction that looksroughly as it does on Facebook). Students stumped about some problem caneasily address it, whatever the hour, if other students are reading atthe same time or soon after. When students are assigned videos, they maynow be able to annotate the timeline, with comments and interactionsfollowing.

Research has shown that students who engage in high levels of meaningfulonline discussion using annotation systems have higher normalizedlearning gain scores than students who participate just to fulfill basicrequirements. Moreover, providing students with incentives to completethe readings thoughtfully and feedback on their annotations helps ensurethat students do the assigned readings on time. Overall, when integratedproperly into the classroom experience, annotations and their evaluationcontribute meaningfully to student learning.

One bar to effective exploitation of online resources and discussionforums is hesitation to initiate a discussion. Students may be chary ofbeing the first to post a comment or question, and annotations thatmerely request clarification or an answer may not contribute to robustdiscussion. Motivating students not only to post annotations but toinitiate meaningful online discourse and debate remains a challenge.Thus, there is a need for techniques and systems for the encouragementand evaluation of annotations during participants' use of onlineeducational resources without significant increase to the workload ofthe instructor and/or other course managers.

SUMMARY

Embodiments of the invention stimulate discussions in online forums andwithin educational resources (e.g., online textbooks) adapted to acceptand publicly display student annotations. In various embodiments,“high-quality” posts, messages, or other annotations (collectively,“annotations”) from one or more concurrent or previous classes orsections are used as seeds to promote productive, generative discussionsin another class or section. As used herein, the term “high-qualityannotations” means annotations that are likely to generate responses andstimulate discussion threads, and “discussion” or “thread” means asequence of online annotations pertaining to a topic or directed towarda portion of a resource (e.g., a paragraph, section or chapter of abook), and a “discussion” may contain one or more “threads.” “Discussionmedia” refers to online platforms where participants may postannotations, e.g., a discussion board or annotatable portion of anonline textbook. Research has shown that seeding discussions withhigh-quality questions from a discussion from a previous class orsection both improves the overall quality of the annotations andpromotes a higher proportion of generative and argumentative discussionthreads. Generative and argumentative discussions have been shown to bethe most effective in promoting learning. In accordance herewith,high-quality annotations are detected automatically, and onceidentified, are used to seed discussions at the class level and/or thesection level.

High-quality annotations may be selected historically, i.e., fromearlier classes and/or earlier and/or concurrent sections using aparticular resource, by simple ranking i.e., examining discussionlengths and, for a particular topic, resource, or portion of a resource,identifying threads that exceed a length threshold (or which are thelongest for the particular topic, resource or resource portion). Theoriginal annotation that stimulated the discussion is then selected.Thus, high-quality annotations from a previous session of a class may beused to seed discussions in a current session.

High-quality annotations may also be selected on a current basis using amachine-learning model optimized to predict and select the annotationsmost likely to generate good, informative discussion. For example,high-quality annotations may be identified in one section and importedinto the discussion media of other sections within the same class; as aresult, different sections may “cross-pollinate” each other so that allstudents benefit from high-quality annotations pertaining to the sameresource used across sections. It has been found that limiting the sizeof the online discussion to 20-40 students optimizes both the qualityand length of the discussion. In some embodiments, classes areautomatically divided into smaller sections of students for purposes ofdiscussion. To standardize the discussion experience across sections, anautomatic seeding approach extracts features from annotations found tolead to long and generative discussions in one class or section and,using these features, builds a model to identify targeted seedannotations made in a particular section that are likely to generategood, informative discussion threads across all sections. Becausesections need not have organizational or administrative significanceoutside the operation of the invention, they may be monitored for theirsuccess in engaging in discussions, and students may be grouped intosections that better optimize student engagement, collaboration, andlearning.

in various embodiments, the model identifies seed annotations before thethreads have emerged and, in real-time, automatically embeds these intodiscussion media in the other sections of the class. This approachprovides the benefit of smaller discussion sections without losing thebenefit of insightful annotations made in another section. The modelused to predict and select the best seed annotations may be iterative,adaptive, and not based on predetermined features. The model may employa dynamic feature-extraction process that changes based on thesuccessful identification of seeds from one portion of a resource to thenext. The model may evaluate the success of each seed in stimulatinggenerative threads by analyzing both the length and average quality ofthe thread that ultimately emerge from each seed, with quality definedabove and scored as described below. Based on the success of the seedsin each resource or resource portion, feature extraction may be refinedso that more predictive features are used with each successive resourceor resource portion. For example, feature extraction may be refined fromone chapter to the next so that more predictive features are used tochoose better seeds. The chosen seeds are automatically embedded i.e.,introduced into discussion media or as postings to an online educationalresource into all sections of the course simultaneously so that allstudents benefit from participation in the generative threads thatemerge

Embodiments of the invention may also analyze discussions inclass-related online forums to identify likely sources of studentconfusion. Particularly when student annotations relate to an onlineeducational resource, embodiments of the invention may assemble a reportwith clickable links to the annotations and/or pertinent portions of theresource. In various embodiments, the invention utilizes the identified“high-quality” annotations, which are likely to generate responses andstimulate discussion threads, as suggestive of areas of studentconfusion.

As utilized herein, the term “annotation” refers to any feedbacksupplied by a student in response to and/or associated with aneducational resource. Annotations may include, for example, posts and/ormessages in electronic discussion forums, answers to embedded questions,comments related to specific passages of the resource, or both. Asutilized herein, the term “class” refers to a gathering of “users,”“participants,” or “students” led by one or more “instructors.”Participants need not be in the same room as each other or theinstructor, so classes encompass distance learning situations. Inaddition, participants need not be students; they might be employeesparticipating in a corporate training event or workshop participantsattending an educational workshop. Accordingly, the terms “participant”and “student” are used interchangeably herein, it being understood thatthe utility of the invention is not limited to students in classroomenvironments. In addition, the term “instructor” used herein is notlimited to a teacher or a professor in the classroom; the “instructor”may be a facilitator in a corporate event or in any group pursuing apedagogical or intellectual endeavor.

In an aspect, embodiments of the invention feature a method of improvingonline discussions in connection with an educational resource providedto students over network-connected devices. In a step (a), aninteractive educational resource is distributed over a network to aplurality of student devices. The student devices are associated withstudents currently enrolled in a class utilizing the educationalresource. In a step (b), an online discussion for receiving and makingvisible, to student devices assigned to a discussion group, annotationsconcerning the educational resource received by the discussion serverfrom the student devices assigned to the discussion group is hosted at adiscussion server. In a step (c), annotations are computationallyanalyzed to identify high-quality annotations likely to generateresponses and stimulate discussion threads. In a step (d), theidentified annotations are made visible to student devices associatedwith students who are not assigned to the discussion group.

Embodiments of the invention may include one or more of the following inany of a variety of combinations. Prior to step (c), the method mayinclude (i) receiving an initial set of annotations at the discussionserver, each of the initial set of annotations having a discussionthread associated therewith, wherein at least a portion of the initialset of annotations constitutes a training set, (ii) extracting portionsof annotations within the training set, thereby producing a plurality ofseed features, and (iii) computationally deriving, from the seedfeatures, one or more evaluation features predictive of thread lengthsof discussion threads associated with annotations in the training set.Step (c) may include, consist essentially of, or consist of using amachine-learning model to predict a thread length (and/or one or moreother quality metrics) associated with each annotation based on the oneor more evaluation features. The model may be predictive in accordancewith a prediction algorithm and may be generated by steps including,consisting essentially of, or consisting of (i) dividing the initial setof annotations into the training set and a testing set, each of thetraining set and the testing set comprising a plurality of annotationsand thread lengths (and/or one or more other quality metrics) associatedtherewith, and (ii) identifying the one or more evaluation featuresbased on predictive reliability in accordance with the predictionalgorithm. Thread lengths (and/or one or more other quality metrics) forone or more annotations within the testing set may be computationallypredicted based on the one or more evaluation features. Parameters ofthe model may be adjusted based on the predictions, for example, priorto computationally analyzing annotations not within the testing set ortraining set to identify high-quality annotations. The predictionalgorithm may include, consist essentially of, or consist of aclassification tree. The prediction algorithm may include, consistessentially of, or consist of a random forest. The random forest mayinclude, consist essentially of, or consist of a plurality of regressiontrees. Producing the plurality of seed features may include, consistessentially of, or consist of applying natural-language processing toannotations within the training set.

The discussion server may host a plurality of simultaneous discussionseach visible only to a discussion group including, consistingessentially of, or consisting of a subset of the students enrolled inthe class. The annotations may be analyzed within at least onediscussion group and identified annotations within one discussion groupmay be made visible to student devices associated with students who arein at least one of the other discussion groups. The discussion group maycorrespond to a first session of the class and at least some of thestudents who are not in the discussion group may be enrolled in asecond, subsequent session of the class. The method may include, afterstep (c), (i) computationally identifying clusters of high-qualityannotations relating to the same portion or related portions of theeducational resource, (ii) for each cluster, extracting and summarizingtext from the annotations indicative of a topic to which the annotationsrelate, and (iii) combining, in an electronically represented document,the extracted and summarized text and (a) at least some of theannotations and the portion or portions of the educational resource or(b) clickable links thereto. The text from each of the clusters may berepresented in the document in the form of a panel. The method mayinclude, after step (d), redefining the discussion group to include oneor more students not assigned to the discussion group in step (b).

In another aspect, embodiments of the invention feature an educationalsystem that includes, consists essentially of, or consists of aplurality of student devices for executing an interactive educationalresource received over a network, a student database, a resource serverin electronic communication with the student devices, a discussionserver, and an analysis module. The student devices are configured toreceive student annotations associated with the educational resource andtransmit at least some of the annotations to the discussion server. Theresource server includes, consists essentially of, or consists of acommunication module. The resource server is configured to make theresource available to student devices associated with students enrolledin a class. The discussion server is in electronic communication withthe student devices. The discussion server receives and makes visible,to student devices assigned to a discussion group in the studentdatabase, annotations concerning the educational resource received fromthe student devices assigned to the discussion group. The analysismodule computationally analyzes annotations to identify high-qualityannotations likely to generate responses and stimulate discussionthreads. The discussion server is configured to make the identifiedannotations visible to student devices associated with students who arenot assigned to the discussion group.

Embodiments of the invention may include one or more of the following inany of a variety of combinations. The analysis module may be configuredto (i) extract portions of annotations within a training set ofannotations, thereby producing a plurality of seed features, and (ii)computationally derive, from the seed features, one or more evaluationfeatures predictive of thread lengths of discussion threads (and/or oneor more other quality metrics) associated with annotations in thetraining set. The analysis module may use a machine-learning model topredict a thread length (and/or one or more other quality metrics)associated with each annotation based on the one or more evaluationfeatures. The model may be predictive in accordance with a predictionalgorithm and may be generated by steps including, consistingessentially of, or consisting of (i) dividing an initial set ofannotations into the training set and a testing set, each of thetraining set and the testing set comprising a plurality of annotationsand thread lengths (and/or one or more other quality metrics) associatedtherewith, and (ii) identifying the one or more evaluation featuresbased on predictive reliability in accordance with the predictionalgorithm. The analysis module may be configured to (i) computationallypredict, based on the one or more evaluation features, thread lengths(and/or one or more other quality metrics) for one or more annotationswithin the testing set, and adjust parameters of the model based on thepredictions. The prediction algorithm may include, consist essentiallyof, or consist of a classification tree. The prediction algorithm mayinclude, consist essentially of, or consist of a random forest. Therandom forest may include, consist essentially of, or consist of aplurality of regression trees. The analysis module may be configured toproduce the plurality of seed features by applying natural-languageprocessing to annotations within the training set.

The discussion server may host a plurality of simultaneous discussionseach visible only to a discussion group including, consistingessentially of, or consisting of a subset of the students enrolled inthe class. The analysis module may be configured to analyze annotationswithin at least one discussion group. The discussion server may makeidentified annotations within one discussion group visible to studentdevices associated with students who are in at least one of the otherdiscussion groups. The discussion group may correspond to a firstsession of the class and at least some of the students who are not inthe discussion group may be enrolled in a second, subsequent session ofthe class. The analysis module may be configured to (i) computationallyidentify clusters of high-quality annotations relating to the sameportion or related portions of the educational resource, (ii) for eachcluster, extract and summarize text from the annotations indicative of atopic to which the annotations relate, and (iii) combine, in anelectronically represented document, the extracted and summarized textand (a) at least some of the annotations and the portion or portions ofthe educational resource or (b) clickable links thereto. The text fromeach of the clusters may be represented in the document in the form of apanel.

In yet another aspect, embodiments of the invention feature a method ofidentifying and summarizing subject matter for improving discussions inconnection with an educational resource provided to students overnetwork-connected devices. In a step (a), an interactive educationalresource is distributed over a network to a plurality of studentdevices. The student devices are associated with students currentlyenrolled in a class utilizing the educational resource. In a step (b),an online discussion for receiving and making visible, to studentdevices assigned to a discussion group, annotations concerning theeducational resource received by the discussion server from the studentdevices assigned to the discussion group is hosted at a discussionserver. In a step (c), annotations are computationally analyzed toidentify high-quality annotations likely to generate responses andstimulate discussion threads. In a step (d), clusters of high-qualityannotations relating to the same portion or related portions of theeducational resource are computationally identified. In a step (e), foreach cluster, text from the annotations indicative of a topic to whichthe annotations relate is extracted and/or summarized. In a step (f),the extracted and/or summarized text and (i) at least some of theannotations and the portion or portions of the educational resource or(ii) clickable links thereto are combined in an electronicallyrepresented document.

Embodiments of the invention may include one or more of the following inany of a variety of combinations. The method may include, prior to step(c), (i) receiving an initial set of annotations at the discussionserver, each of the initial set of annotations having a discussionthread associated therewith, wherein at least a portion of the initialset of annotations constitutes a training set, (ii) extracting portionsof annotations within the training set, thereby producing a plurality ofseed features, and (iii) computationally deriving, from the seedfeatures, one or more evaluation features predictive of thread lengthsof discussion threads (and/or one or more other quality metrics)associated with annotations in the training set. Step (c) may include,consist essentially of, or consist of using a machine-learning model topredict a thread length (and/or one or more other quality metrics)associated with each annotation based on the one or more evaluationfeatures. The model may be predictive in accordance with a predictionalgorithm and may be generated by steps including, consistingessentially of, or consisting of (i) dividing the initial set ofannotations into the training set and a testing set, each of thetraining set and the testing set comprising a plurality of annotationsand thread lengths (and/or one or more other quality metrics) associatedtherewith, and (ii) identifying the one or more evaluation featuresbased on predictive reliability in accordance with the predictionalgorithm. Thread lengths (and/or one or more other quality metrics) forone or more annotations within the testing set may be computationallypredicted based on the one or more evaluation features. Parameters ofthe model may be adjusted based on the predictions, for example, priorto computationally analyzing annotations not within the testing set ortraining set to identify high-quality annotations. The predictionalgorithm may include, consist essentially of, or consist of aclassification tree. The prediction algorithm may include, consistessentially of, or consist of a random forest. The random forest mayinclude, consist essentially of, or consist of a plurality of regressiontrees. Producing the plurality of seed features may include, consistessentially of, or consist of applying natural-language processing toannotations within the training set.

The text from each of the clusters may be represented in the document inthe form of a panel. After step (c), the identified annotations may bemade visible to student devices associated with students who are notassigned to the discussion group. The discussion server may host aplurality of simultaneous discussions each visible only to a discussiongroup including, consisting essentially of, or consisting of a subset ofthe students enrolled in the class. The annotations may be analyzedwithin each discussion group. One or more identified annotations withinone discussion group may be made visible to student devices associatedwith students who are (i) in one or more of the other discussion groups,and/or (ii) not assigned to the discussion group. The discussion groupmay correspond to a first session of the class. The students who are notassigned to the discussion group may be enrolled in a second, subsequentsession of the class.

In another aspect, embodiments of the invention feature an educationalsystem that includes, consists essentially of, or consists of aplurality of student devices for executing an interactive educationalresource received over a network, a student database, a resource serverin electronic communication with the student devices, a discussionserver, and an analysis module. The student devices are configured toreceive student annotations associated with the educational resource andtransmit at least some of the annotations to the discussion server. Theresource server includes, consists essentially of, or consists of acommunication module. The resource server is configured to make theresource available to student devices associated with students enrolledin a class. The discussion server is in electronic communication withthe student devices. The discussion server receives and makes visible,to student devices assigned to a discussion group in the studentdatabase, annotations concerning the educational resource received fromthe student devices assigned to the discussion group. The analysismodule is configured to (i) computationally analyze annotations toidentify high-quality annotations likely to generate responses andstimulate discussion threads, (ii) computationally identify clusters ofhigh-quality annotations relating to the same portion or relatedportions of the educational resource, (iii) for each cluster, extractand/or summarize text from the annotations indicative of a topic towhich the annotations relate, and (iv) combine, in an electronicallyrepresented document, the extracted and/or summarized text and (a) atleast some of the annotations and the portion or portions of theeducational resource or (b) clickable links thereto.

Embodiments of the invention may include one or more of the following inany of a variety of combinations. The analysis module may be configuredto (i) extract portions of annotations within a training set ofannotations, thereby producing a plurality of seed features, and (ii)computationally derive, from the seed features, one or more evaluationfeatures predictive of thread lengths of discussion threads (and/or oneor more other quality metrics) associated with annotations in thetraining set. The analysis module may use a machine-learning model topredict a thread length (and/or one or more other quality metrics)associated with each annotation based on the one or more evaluationfeatures. The model may be predictive in accordance with a predictionalgorithm and may be generated by steps including, consistingessentially of, or consisting of (i) dividing an initial set ofannotations into the training set and a testing set, each of thetraining set and the testing set comprising a plurality of annotationsand thread lengths (and/or one or more other quality metrics) associatedtherewith, and (ii) identifying the one or more evaluation featuresbased on predictive reliability in accordance with the predictionalgorithm. The analysis module may be configured to (i) computationallypredict, based on the one or more evaluation features, thread lengths(and/or one or more other quality metrics) for one or more annotationswithin the testing set, and adjust parameters of the model based on thepredictions. The prediction algorithm may include, consist essentiallyof, or consist of a classification tree. The prediction algorithm mayinclude, consist essentially of, or consist of a random forest. Therandom forest may include, consist essentially of, or consist of aplurality of regression trees. The analysis module may be configured toproduce the plurality of seed features by applying natural-languageprocessing to annotations within the training set.

The discussion server may be configured to make the identifiedannotations visible to student devices associated with students who arenot assigned to the discussion group. The discussion server may host aplurality of simultaneous discussions each visible only to a discussiongroup including, consisting essentially of, or consisting of a subset ofthe students enrolled in the class. The analysis module may beconfigured to analyze annotations within each discussion group. Thediscussion server may make identified annotations within one discussiongroup visible to student devices associated with students who are (i) inone or more of the other discussion groups, and/or (ii) not assigned tothe discussion group. The discussion group may correspond to a firstsession of the class. The students who are not assigned to thediscussion group may be enrolled in a second, subsequent session of theclass.

These and other objects, along with advantages and features of thepresent invention herein disclosed, will become more apparent throughreference to the following description, the accompanying drawings, andthe claims. Furthermore, it is to be understood that the features of thevarious embodiments described herein are not mutually exclusive and mayexist in various combinations and permutations. As used herein, theterms “approximately” and “substantially” mean ±10%, and in someembodiments, ±5%. The term “consists essentially of” means excludingother materials that contribute to function, unless otherwise definedherein.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to the sameparts throughout the different views. Also, the drawings are notnecessarily to scale, emphasis instead generally being placed uponillustrating the principles of the invention. In the followingdescription, various embodiments of the present invention are describedwith reference to the following drawings, in which:

FIG. 1 is a schematic depiction of an educational environment inaccordance with various embodiments of the invention;

FIG. 2 is a block diagram of an educational server or system utilized inaccordance with various embodiments of the invention; and

FIG. 3 is a flowchart of a technique of improving online and/or offlinediscussions in connection with an educational resource in accordancewith various embodiments of the invention.

DETAILED DESCRIPTION

FIG. 1 illustrates an exemplary educational environment 100 inaccordance with embodiments of the present invention. As shown, withinthe environment 100, communication is established, via a network 110,among an instructor 120 utilizing an instructor device 130, variousstudents 140 each utilizing a student device 150, one or more optionalgraders 160 each utilizing a grading device 170, and an educationalsystem or server 180. Graders 160 may include or consist essentially of,for example, (1) staff graders, i.e., teaching assistants hand-gradingstudent annotations with a research-based rubric, (2) peer graders who,in the process of learning about scoring rubrics used to evaluateannotations, score a subset of their peers' annotations through acalibration grading exercise (thus, one or more of the graders 160 mayalso be a student 140), and/or (3) dedicated human graders not enrolledin the class.

The network 110 may include or consist essentially of, for example, theInternet and/or one or more local-area networks (LANs) or wide-areanetworks (WANs). The terms “student device,” “instructor device,” and“grading device” as used herein broadly connote any electronic device orsystem facilitating wired and/or wireless bi-directional communications,and may include computers (e.g., laptop computers and/or desktopcomputers), handheld devices, or other personal communication devices.Handheld devices include, for example, smart phones or tablets capableof executing locally stored applications and supporting wirelesscommunication and data transfer via the Internet or the publictelecommunications infrastructure. Smart phones include, for example,IPHONES (available from Apple Inc., Cupertino, Calif.), BLACKBERRIES(available from RIM, Waterloo, Ontario, Canada), or any mobile phonesequipped with the ANDROID platform (available from Google Inc., MountainView, Calif.); tablets, such as the IPAD and KINDLE FIRE; and personaldigital assistants (PDAs). The bi-directional communication and datatransfer may take place via, for example, one or more of cellulartelecommunication, a Wi-Fi LAN, a point-to-point Bluetooth connection,and/or an NFC communication.

FIG. 2 depicts a more detailed schematic of the server 180, whichincludes or consists essentially of a general-purpose computing devicewhose operation is directed by a computer processor, i.e., centralprocessing unit (CPU) 200. The server 180 includes a network interface205 that facilitates communication over the network 110, using hypertexttransfer protocol (HTTP) or other suitable protocols. For example, thenetwork interface 205 may include or consist essentially of one or morehardware interfaces enabling data communication via network 110, as wellas a communications module for sending, receiving, and routing suchcommunications within server 180 (e.g., via system bus 210). The server180 further includes a bi-directional system bus 210, over which thesystem components communicate, a main (typically volatile) system memory215, and a non-volatile mass storage device (such as one or more harddisks and/or optical storage units) 220, which may contain resources,such as digital textbooks and/or other educational resources, that maybe delivered to the student devices 150.

The main memory 215 contains instructions, conceptually illustrated as agroup of modules, which control the operation of the CPU 200 and itsinteraction with the other hardware components. An operating system 225directs the execution of low-level, basic system functions such asmemory allocation, file management and operation of mass storage devices220. The operating system 225 may be or include a variety of operatingsystems such as Microsoft WINDOWS operating system, the Unix operatingsystem, the Linux operating system, the Xenix operating system, the IBMAIX operating system, the Hewlett Packard UX operating system, theNovell NETWARE operating system, the Sun Microsystems SOLARIS operatingsystem, the OS/2 operating system, the BeOS operating system, theMACINTOSH operating system, the APACHE operating system, an OPENSTEPoperating system or another operating system of platform.

A resource-management module 230 is responsible for, e.g., allowingproperly authenticated students 140 to access privileged educationalresources via their devices 150, and for monitoring the students'interactions with these resources. The resource-management module 230may also control and facilitate access to educational resources for theinstructor 120 via the instructor device 130 and/or for the graders 160via grading devices 170. It should be understood that resources providedto the student devices 150 need not reside physically within the server180; the resource-management module 230 may obtain resources from otherservers, or direct other servers (e.g., an educational publisher'sserver) to provide resources to student devices. It should further beunderstood that the access-control functions of the resource-managementmodule 230 are well known to those skilled in the art of onlineeducational platforms and, more generally, to access control forresources available online or via a private network.

In accordance with embodiments of the invention, an analysis engine (or“analysis module”) 235 monitors student annotations and analyzesannotations to identify high-quality annotations. The server 180 mayalso maintain or have access to a student database 240 containingcontact information for each student, including email addresses, phonenumbers (e.g., to which text messages may be sent). The student database240 may also maintain rosters of classes, sections, and students withineach class section. In various embodiments of the invention, the server180 may also incorporate a discussion hosting server 245 that supports adiscussion platform and makes this available to students 140 via theirdevices 150. The discussion platform may be a server-hosted discussionboard that operates autonomously, in the manner of a social-mediaplatform, or may be associated with resources 220 in the manner ofdiscussion boards maintained by online educational platforms such as edXor COURSERA. For example, server 245 may perform the functions ofresource-management module 230 and facilitate access to resources 220that have annotation fields into which students 140 may enter commentsthat server 245 organizes as annotation threads (or “discussionthreads”). Server 245 may be part of the main server 180 or may be aseparate device.

As mentioned previously, the analysis engine 235 monitors studentannotations entered into the platform provided by hosting server 245,and analyzes these to identify high-quality annotations. This analysismay take place on a historical basis, e.g., by ranking annotationsentered during a previous class session, or on a current basis, e.g., bypredicting which current posts are likely to be high-quality. Each classsection may interact with a separate discussion platform. When ahigh-quality post is identified, for example, from a prior class,hosting server 245 may post it to all relevant platforms (i.e., acrossall sections of the current class) to stimulate discussion. Similarly,when a high-quality post is identified in a section of the current classsession, hosting server 245 may post it to the discussion platforms ofall other sections.

The server 180 may also include, in various embodiments of theinvention, a repository or database 250 that stores various reportsrelated to the interactions of students 140, the instructor 120, and/orgraders 160 with the resources 220 (and/or with content related thereto,such as student annotations). For example, the repository 250 may storegrade reports generated by graders 160 or reports for the instructor 120based on and/or highlighting questions, comments, and/or annotationsgenerated by the students 140. For example, such reports may includelinks to annotations stored on the discussion server 245.

The analysis engine 235 may use high-quality annotations as the basisfor generating interactive, editable “confusion reports” —i.e.,electronic documents that highlight and summarize what the studentsfound the most confusing in the educational resource (or a portionthereof). The confusion reports may be stored in the repository 250 aselectronic documents in any suitable format (e.g., as WORD documents,PDF documents, HTML files, etc.) for convenient retrieval andinteractivity. For example, confusion reports may include links toannotations stored on discussion server 245.

Analysis engine 235 may also parse identified annotations to locate keyterms indicating the source of confusion and/or creating brief,automated summaries of a cluster of high-quality annotations relating toa particular topic.

FIG. 3 depicts a method 300 for improving online discussions inconnection with an educational resource in accordance with variousembodiments of the present invention. In step 305, an educationalresource or a portion thereof (e.g., from storage 220) is electronicallydistributed to one or more student devices 150 via network 110. Thestudent devices 105 to which the resource is distributed may beassociated with, for example, students 140 currently enrolled in a classutilizing the resource. During use of the educational resource (e.g.,reading of one or more passages in an electronic textbook and/oranswering questions related to the resource) by the students 140, thestudents 140 may supply annotations related to the resource via theirstudent devices 150. In step 310, an online discussion is hosted at thediscussion server 245 in order to receive the annotations and make themvisible to at least a subset of students 140 utilizing the resource(e.g., students 140 enrolled in the class). For example, the onlinediscussion may be configured as a plurality of discussion threadspertaining to various topics relevant to the educational resource. Theclass may be split into multiple different discussion groups eachcontaining a subset of students 140 enrolled in the class, and eachdiscussion group may have a dedicated online discussion and/or set ofdiscussion threads dedicated and visible to only that discussion group.In various embodiments of the invention, a “discussion group” maycorrespond to a first session of a class utilizing the educationalresource, and other discussion groups (or students not in the discussiongroup) correspond to subsequent sessions of the same class.

In step 315, the annotations within the online discussion (or portionthereof) dedicated to one of the discussion groups are computationallyanalyzed by analysis engine 235 to identify “high-quality” annotations,i.e., annotations likely to generate responses and thereby stimulatediscussion threads, and/or annotations having high quality as evaluatedin accordance with U.S. Provisional Application No. 62/261,398, filed onDec. 1, 2015, and in U.S. Patent Application Ser. No. 15/______,______,entitled “AUTOMATED GRADING FOR INTERACTIVE LEARNING APPLICATIONS,”filed concurrently herewith, the entire disclosure of each of which isincorporated by reference herein. For example, high-quality annotationsmay be annotations that result in long discussion threads (i.e.,discussion threads having more than a predetermined number ofannotations, or simply one or more of the longest threads in the onlinediscussion) and/or discussion threads involving many different students(i.e., discussion threads eliciting annotations from more than apredetermined number of students in the discussion group, or simply oneor more of the threads in the online discussions having the highestnumber of participating students). Of course, other indicia ofdiscussion quality (such as average word length, word sophistication asindicated by, e.g., a statistical metric such as term frequency/inversedocument frequency (TF/IDF)) may be utilized alternatively or inaddition.

In step 320, once the high-quality annotations made by the discussiongroup are identified, the discussion server 245 makes those annotationsvisible to students not assigned to the discussion group. For example,the selected annotations may be utilized as “seed annotations” in therelevant portion(s) of the educational resource for one or morediscussion groups of students 140 currently enrolled in the same classand/or for discussion groups corresponding to students 140 enrolled inone or more subsequent (i.e., later in time) sessions of the class. Inother exemplary embodiments, the selected annotations may be displayedto students enrolled in different classes and/or different courses atthe same educational institution or at or educational institutionsdifferent from the institution from which the annotations arose.Annotations made visible to students not assigned to the discussiongroup may be anonymized (i.e., any information identifying thestudent(s) generating the annotations may be removed) before suchannotations are made visible to other students.

In various embodiments of the invention, annotations may also beoptionally analyzed to generate a predictive model that identifiesannotations as high-quality annotations even before all or portions ofdiscussion thread(s) associated therewith emerge via student discussion.For example, annotations (e.g., those at or near the beginning ofemerging discussion threads) may be analyzed using a machine-learningmodel to identify annotations and/or portions thereof that arepredictive of high-quality discussion. In various embodiments, the modelmay utilize conventional natural-language processing techniques such asstemming, stop-word removal and/or part-of-speech tagging.

The machine-learning model may be trained utilizing high-qualityannotations identified in step 315 and be subsequently utilized topredict whether new student annotations are high-quality even beforediscussions associated therewith has been continued or completed. Forexample, some or all of the high-quality annotations identified in step315 may be used as a training set for a text-analytic regressionprocedure (e.g., logistic regression, classification tree, random forestclassifier, etc.) that constitutes the machine-learning model. Moregenerally, the machine-learning model may be any suitable analyticframework for analyzing text and making predictions based on a trainingset, including classification and regression trees (CART), neuralnetworks, or other suitable framework. Machine-learning models arewell-characterized in the art and may be implemented without undueexperimentation.

For example, in various embodiments, the analysis engine 235 utilizes arandom forest classifier as the basis for the machine-learning model. Inone embodiment, high-quality annotations form a training set, and thediscussion thread quality (e.g., thread length or other discussionquality metric) associated with each annotation is determined from theexisting discussion. The thread quality values serve as category labelsfor the machine-learning model. Features from the high-qualityannotations may be extracted from the text and one or, more preferably,an ensemble of classifiers is used to fit the model to predict thecategory labels from the annotation features. The annotation features,along with permutations and combinations thereof, form a set ofcandidate evaluation features.

In step 325 depicted in FIG. 3, the evaluation features with sufficientpredictive reliability against the training set may be selected for usein the model. The predictive reliability of a feature may be deemedsufficient, for example, based on standard error, t value, p value oranother statistical metric, for example, a minimum p value required foran annotation feature to qualify as an evaluation feature set at astandard level of 0.01 or less. (The p value reflects the probabilitythat the feature has no predictive value.) Typically, the training setwill have 100 or more entries each reflecting a thread length and/orthread-quality metric associated with an annotation.

In step 330, following creation of the model and feature selection usingthe training set, the performance of the model may be evaluated using atesting set composed of other high-quality annotations identified instep 315 (i.e., having known thread length and/or quality metricsassociated therewith). For example, the false positive and falsenegative predictions obtained against the testing set may be used todetect overfitting, identify and prune features exhibitingmulticollinearity, and set a classification threshold that produces adesired level of sensitivity (true positive rate) and specificity (truenegative rate). A set of evaluation features that produces predictedthread lengths and/or quality metrics having values different from theactual, known thread lengths and/or quality metrics by less than apredetermined threshold amount (e.g., ±10%, ±5%, ±2%, etc.) may beselected for subsequent predicts of high-quality annotations. In thismanner, the quality of annotations may be predicted even beforediscussions associated therewith are commenced or completed, and suchannotations may also be made visible to students in other discussiongroups even before discussions associated therewith in the originatingdiscussion group are complete.

As known to those of skill in the art, random forest classifiers operatevia the construction of several decision trees based on the trainingset, and output the class that is the mode of the classes(classification) or mean prediction (regression) of the individualtrees—thereby correcting for potential overfitting of the training set.Other text-analytic techniques, as noted above, may be utilized by theanalysis engine 235 to determine evaluation features that predict threadlengths and/or quality metrics associated with the annotations in thetesting group. As shown in FIG. 3, this predictive procedure may berepeated one or more times to refine the machine-learning model toinclude an ensemble of evaluation features (i.e., classifiers extractedfrom the seed features) that most accurately predicts the thread lengthsand/or quality metrics associated with annotations in the testing set,or an ensemble of evaluation features that at least predicts the threadlengths and/or quality metrics to within a desired level of accuracy.

After development of the model, in step 335, new student annotationsfrom a discussion group are received by the server 180. For example,such annotations may include annotations related to a subsequentexercise or received from a different discussion group of students 140than those whose annotations were utilized to define the evaluationfeatures and thus generate the machine-learning model. Using theevaluation features, the analysis engine 235 predicts the quality of thenew annotations in step 340. As shown in FIG. 3, any high-qualityannotations identified in step 340 may also be made visible to otherstudents not within the discussion group from which the newly identifiedannotations were generated.

In various embodiments, one or more discussion groups may be redefinedbased upon annotations produced within one discussion group and madevisible to students within one or more other discussion groups. Forexample, one or more such annotations may elicit high-qualityannotations from one or more students in different discussion groups,and such students may be assigned to the same discussion group forsubsequent lessons. Such embodiments of the invention may thus identifyand group students that respond particularly well to annotations ofstudents not initially in their discussion group(s).

in various embodiments of the invention, the high-quality annotationsidentified in step 315 may be utilized to highlight for the instructor120 topics or portions of the educational resource found to be confusingby the students 140 and/or which elicited the most discussion by thestudents 140. For example, the annotations may be ranked and clustered,and presented in a manner that facilitates convenient access to theposts and the source material to which they relate. For example,embodiments of the invention may generate confusion reports thathighlight and summarize what the students 140 found the most confusingin the reading, thereby helping the instructor prepare and use classroomtime wisely. More broadly, the confusion report may contain allhigh-quality annotations (or an arbitrary number of them), whether theyindicate confusion or correct understanding.

Referring back to FIG. 3, in step 345, the analysis engine 235 maycomputationally analyze the high-quality annotations in order toidentify clusters relating to the same portion or related portions ofthe educational resource (e.g., relating to the same topic or relatedtopics). For example, the analysis engine 235 may apply natural-languageprocessing to the high-quality annotations and/or to the portions of theresource with which they are associated in order to cluster theannotations on the basis of subject matter. Thus, a statistical metricsuch as TF/IDF may be applied to the annotations to identifysubject-specific vocabulary that is used to identify and cluster relatedannotations. Alternatively or in addition, the annotations may beclustered based on the location(s) of the annotations within theresource; for example, online discussion boards may be partitioned bycourse segment and/or topic, and annotations within each partition forma duster.

in step 350, the analysis engine 235 may extract and/or summarize textfrom the annotations of each cluster that is indicative of the topic towhich the annotations in each cluster relate. To perform this step, theanalysis engine 235 may utilize topic modeling with a non-negativematrix factorization algorithm. Such algorithms are well-known and maybe implemented without undue experimentation (see, e.g., Cichocki etal., Nonnegative Matrix and Tensor Factorizations (John Wiley & Sons2009), the entire disclosure of which is incorporated by referenceherein).

in step 355, the analysis engine 235 produces one or more confusionreports based on the clusters of annotations identified in step 345 andthe text produced in step 350. For example, the confusion report mayinclude a machine-generated summary of the high-quality annotations(e.g., questions students asked about an online text), and may furtherinclude extracted, intact portions of the pertinent resource. Thesummary may be in the form of short phrases or descriptions thatsummarize the concept or point of confusion. Clickable links to theresource and/or annotations may also be provided. In some embodiments,the confusion report includes figures and formatting to reproduce the“look and feel” of the resource.

In some embodiments, clusters of annotations and ancillary material(i.e., portions of annotations and/or material drawn from theeducational resource, as well as, in some cases, links thereto) relatingto a single topic or resource portion are presented in the form of apanel, several of which may be contained in a single confusion report.The instructor may be able to edit the confusion report, e.g., to addtext or figures, drag panels around, turn them into thumbnails, andannotate them further in order to prepare for efficient use of classroomtime.

A panel may include links to the annotations, and may sync, inreal-time, with recently added annotations. If a student 140 adds a newannotation that falls into one of the categories summarized in thereport, it (or a link to it) may automatically appear in the relevantpanel. Each panel may be presented in its full format or in a thumbnailformat, as well as with an automated text summary of the content of theannotations on that panel and similar student annotations elsewhere.This format informs the instructor not only of areas where students haveengaged significantly with the material (out of confusion or otherwise),but also allows the instructor to go into the classroom prepared toengage with his students by calling on them by name and referring backto their specific comments. The use of high-quality annotations toproduce the confusion report will tend to improve the quality of theclass conversation and the instruction provided by the instructor.

The resource-management module 230 and analysis engine 235 (and, e.g., acommunications module within or corresponding to network interface 205)may be implemented by computer-executable instructions, such as programmodules, that are executed by a conventional computer. Generally,program modules include routines, programs, objects, components, datastructures, etc. that performs particular tasks or implement particularabstract data types. Those skilled in the art will appreciate thatembodiments of the invention may be practiced with various computersystem configurations, including multiprocessor systems,microprocessor-based or programmable consumer electronics,minicomputers, mainframe computers, and the like. Embodiments of theinvention may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer-storage media including memory storage devices.

Any suitable programming language may be used to implement without undueexperimentation the analytical functions described above.Illustratively, the programming language used may include assemblylanguage, Ada, APL, Basic, C, C++, C*, COBOL, dBase, Forth, FORTRAN,Java, Modula-2, Pascal, Prolog, Python, REXX, and/or JavaScript forexample. Regression-based models (e.g., logistic regression,classification trees and random forests) are readily implemented in theR programming language without undue experimentation (using, e.g., therpart and randomForest libraries), and neural networks may beimplemented in Python or MATLAB. Further, it is not necessary that asingle type of instruction or programming language be utilized inconjunction with the operation of embodiments of the invention. Rather,any number of different programming languages may be utilized as isnecessary or desirable.

The server 180 may also include other removable/nonremovable,volatile/nonvolatile computer storage media. For example, a hard diskdrive may read or write to nonremovable, nonvolatile magnetic media. Amagnetic disk drive may read from or writes to a removable, nonvolatilemagnetic disk, and an optical disk drive may read from or write to aremovable, nonvolatile optical disk such as a CD-ROM or other opticalmedia. Other removable/nonremovable, volatile/nonvolatile computerstorage media that may be used in the exemplary operating environmentinclude, but are not limited to, magnetic tape cassettes, flash memorycards, digital versatile disks, digital video tape, solid state RAM,solid state ROM, and the like. The storage media are typically connectedto the system bus through a removable or non-removable memory interface.

The processing units that execute commands and instructions may begeneral-purpose processors, but may utilize any of a wide variety ofother technologies including special-purpose hardware, a microcomputer,mini-computer, mainframe computer, programmed microprocessor,microcontroller, peripheral integrated circuit element, a CSIC(customer-specific integrated circuit), ASIC (application-specificintegrated circuit), a logic circuit, a digital signal processor, aprogrammable logic device such as an FPGA (field-programmable gatearray), PLD (programmable logic device), PLA (programmable logic array),RFID processor, smart chip, or any other device or arrangement ofdevices that is capable of implementing the steps of the processes ofthe invention.

Communication may occur over the Internet, as illustrated, and/or overan intranet, extranet, Ethernet, the public telecommunicationsinfrastructure, or any other system that provides communications. Somesuitable communications protocols may include TCP/IP, UDP, or OSI forexample. For wireless communications, communications protocols mayinclude Bluetooth, Zigbee, IrDa or other suitable protocol. Furthermore,components of the system may communicate through a combination of wiredor wireless paths.

The terms and expressions employed herein are used as terms andexpressions of description and not of limitation, and there is nointention, in the use of such terms and expressions, of excluding anyequivalents of the features shown and described or portions thereof. Inaddition, having described certain embodiments of the invention, it willbe apparent to those of ordinary skill in the art that other embodimentsincorporating the concepts disclosed herein may be used withoutdeparting from the spirit and scope of the invention. Accordingly, thedescribed embodiments are to be considered in all respects as onlyillustrative and not restrictive.

What is claimed is:
 1. A method of identifying and summarizing subjectmatter for improving discussions in connection with an educationalresource provided to students over network-connected devices, the methodcomprising the steps of: (a) distributing an interactive educationalresource over a network to a plurality of student devices, the studentdevices being associated with students currently enrolled in a classutilizing the educational resource; (b) hosting, at a discussion server,an online discussion for receiving and making visible, to studentdevices assigned to a discussion group, annotations concerning theeducational resource received by the discussion server from the studentdevices assigned to the discussion group; (c) computationally analyzingannotations to identify high-quality annotations likely to generateresponses and stimulate discussion threads; (d) computationallyidentifying clusters of high-quality annotations relating to the sameportion or related portions of the educational resource; (e) for eachcluster, extracting and summarizing text from the annotations indicativeof a topic to which the annotations relate; and (f) combining, in anelectronically represented document, the extracted and summarized textand (i) at least some of the annotations and the portion or portions ofthe educational resource or (ii) clickable links thereto.
 2. The methodof claim 1, further comprising, prior to step (c): receiving an initialset of annotations at the discussion server, each of the initial set ofannotations having a discussion thread associated therewith, wherein atleast a portion of the initial set of annotations constitutes a trainingset; extracting portions of annotations within the training set, therebyproducing a plurality of seed features; and computationally deriving,from the seed features, one or more evaluation features predictive ofthread lengths of discussion threads associated with annotations in thetraining set.
 3. The method of claim 2, wherein step (c) comprises usinga machine-learning model to predict a thread length associated with eachannotation based on the one or more evaluation features, the model beingpredictive in accordance with a prediction algorithm and generated bysteps comprising: dividing the initial set of annotations into thetraining set and a testing set, each of the training set and the testingset comprising a plurality of annotations and thread lengths associatedtherewith; and identifying the one or more evaluation features based onpredictive reliability in accordance with the prediction algorithm. 4.The method of claim 3, further comprising the steps of: computationallypredicting, based on the one or more evaluation features, thread lengthsfor one or more annotations within the testing set; and adjustingparameters of the model based on the predictions prior tocomputationally analyzing annotations not within the testing set ortraining set to identify high-quality annotations.
 5. The method ofclaim 3, wherein the prediction algorithm is a classification tree. 6.The method of claim 5, wherein the prediction algorithm is a randomforest comprising a plurality of regression trees.
 7. The method ofclaim 2, wherein producing the plurality of seed features comprisesapplying natural-language processing to annotations within the trainingset.
 8. The method of claim 1, wherein the text from each of theclusters is represented in the document in the form of a panel.
 9. Themethod of claim 1, further comprising, after step (c), making theidentified annotations visible to student devices associated withstudents who are not assigned to the discussion group.
 10. The method ofclaim 1, wherein the discussion server hosts a plurality of simultaneousdiscussions each visible only to a discussion group consisting of asubset of the students enrolled in the class.
 11. The method of claim10, wherein the annotations are analyzed within each discussion groupand identified annotations within one discussion group are made visibleto student devices associated with students who are (i) in one or moreof the other discussion groups, and/or (ii) not assigned to thediscussion group.
 12. The method of claim 11, wherein the discussiongroup corresponds to a first session of the class and the students whoare not assigned to the discussion group are enrolled in a second,subsequent session of the class.
 13. An educational system comprising: aplurality of student devices for executing an interactive educationalresource received over a network, the student devices being configuredto receive student annotations associated with the educational resourceand transmit at least some of the annotations to a discussion server; astudent database; a resource server in electronic communication with thestudent devices, the resource server comprising a communication moduleand being configured to make the resource available to student devicesassociated with students enrolled in a class; a discussion server, inelectronic communication with the student devices, for receiving andmaking visible, to student devices assigned to a discussion group in thestudent database, annotations concerning the educational resourcereceived from the student devices assigned to the discussion group; andan analysis module configured to (i) computationally analyze annotationsto identify high-quality annotations likely to generate responses andstimulate discussion threads, (ii) computationally identify clusters ofhigh-quality annotations relating to the same portion or relatedportions of the educational resource, (iii) for each cluster, extractand summarize text from the annotations indicative of a topic to whichthe annotations relate, and (iv) combine, in an electronicallyrepresented document, the extracted and summarized text and (a) at leastsome of the annotations and the portion or portions of the educationalresource or (b) clickable links thereto.
 14. The system of claim 13,wherein the analysis module is configured to: extract portions ofannotations within a training set of annotations, thereby producing aplurality of seed features; and computationally derive, from the seedfeatures, one or more evaluation features predictive of thread lengthsof discussion threads associated with annotations in the training set.15. The system of claim 14, wherein the analysis module uses amachine-learning model to predict a thread length associated with eachannotation based on the one or more evaluation features, the model beingpredictive in accordance with a prediction algorithm and generated bysteps comprising: dividing an initial set of annotations into thetraining set and a testing set, each of the training set and the testingset comprising a plurality of annotations and thread lengths associatedtherewith; and identifying the one or more evaluation features based onpredictive reliability in accordance with the prediction algorithm. 16.The system of claim 15, wherein the analysis module is configured to:computationally predict, based on the one or more evaluation features,thread lengths for one or more annotations within the testing set; andadjust parameters of the model based on the predictions.
 17. The systemof claim 15, wherein the prediction algorithm is a classification tree.18. The system of claim 17, wherein the prediction algorithm is a randomforest comprising a plurality of regression trees.
 19. The system ofclaim 14, wherein the analysis module is configured to produce theplurality of seed features by applying natural-language processing toannotations within the training set.
 20. The system of claim 13, whereinthe discussion server is configured to make the identified annotationsvisible to student devices associated with students who are not assignedto the discussion group.
 21. The system of claim 13, wherein thediscussion server hosts a plurality of simultaneous discussions eachvisible only to a discussion group consisting of a subset of thestudents enrolled in the class.
 22. The system of claim 21, wherein theanalysis module is configured to analyze annotations within eachdiscussion group, the discussion server making identified annotationswithin one discussion group visible to student devices associated withstudents who are (i) in one or more of the other discussion groups,and/or (ii) not assigned to the discussion group.
 23. The system ofclaim 22, wherein the discussion group corresponds to a first session ofthe class and the students who are not assigned to the discussion groupare enrolled in a second, subsequent session of the class.